Digital image processing method and device

ABSTRACT

A method of processing digital images wherein:  
     a plurality of images of reduced size are generated from a two-dimensional image, said reduced sizes being different from each other,  
     said reduced size images are stored in memory, and  
     a three-dimensional image is generated in real time from a frame provided with unit surfaces and from pixels chosen from said image and said reduced size images as a function of the orientation of a unit surface to which said pixel is to be assigned.

[0001] The present invention relates to processing digital imagesdisplayed on a screen, in particular on a computer or television screen.

[0002] The portable computer and telephone markets are currentlymerging, driven by the expansion of the Internet and interactivetelevision, which call for associated technologies. It is thereforedesirable to be able to display graphical images, such as World Wide Webpages distributed via the Internet or three-dimensional video gameimages, on a standard television screen with a level of visual comfortthat is acceptable to the user. In particular, video game images aresubject to a phenomenon known as “aliasing” which compromises imagequality.

[0003] In a video game image a structure such as a tree, a vehicle or abuilding is generally defined by an array of triangles defining acontour of the structure and a texture at several scales, the scalebeing determined as a function of the orientation of each triangle. In aconventional video game running on a game console, all the textures arepre-stored at all the scales.

[0004] The invention proposes a way of processing images enablinghigh-quality three-dimensional images, suitable for broadcasting to atelevision screen, to be generated from a stream of data received byradio, by cable, from a satellite, etc.

[0005] The invention proposes a way of processing images enablingthree-dimensional images to be generated from a stream of data in realtime.

[0006] A method of processing digital images according to one aspect ofthe invention includes the following steps:

[0007] a plurality of images of reduced size are generated from atwo-dimensional image, said reduced sizes being different from eachother,

[0008] said images of reduced size are transmitted to a 3D acceleratorencapsulated in accordance with a predetermined structure,

[0009] said reduced size images are stored in memory by extracting saidimages from the encapsulation structure, and

[0010] a three-dimensional image is generated in real time from a frameprovided with unit surfaces and from pixels chosen from said image andsaid reduced size images as a function of the orientation of a unitsurface to which said pixel is to be assigned.

[0011] In other words, the preprocessed video is transferred to the 3Drendition operator. All the small images are encapsulated in a standarddigital video signal in accordance with an X-Y placement convention. Thestandard can be of the ITU-RBT.656 type. The capture port of the 3Dprocessor recovers the video and extracts from it the sequence ofencapsulated images, which is stored in memory with an organizationenabling the rendition operator to derive the best possible quality fromthe video source.

[0012] A reduced image is advantageously generated from the whole of ananimated two-dimensional image.

[0013] Said reduced sizes are advantageously in geometrical progression.

[0014] In one embodiment of the invention the scale of the first reducedscale image is reduced by a factor of 2. In one embodiment of theinvention the scale of the nth reduced scale image is reduced by afactor of 2^(n).

[0015] Storage in memory is preferably effected in accordance with aparticular writing order.

[0016] Reading in memory is preferably effected in a particular order.

[0017] Said reduced size images are advantageously formatted inside theformat of said image so that all of said reduced size images occupy lessthan or the same space as said image.

[0018] In one embodiment of the invention the orientation of the unitsurface to which said pixel will be assigned is defined by a vectornormal to said unit surface.

[0019] In one embodiment of the invention the three-dimensional image isstored in memory.

[0020] In one embodiment of the invention the three-dimensional image issent to a display system.

[0021] A central unit can be adapted to determine the orientation of theunit surface to which said pixel is to be assigned. A digital terminalcan be adapted to generate the reduced-scale images in real time inanimated video. An auxiliary unit can be adapted to assign a unitsurface a pixel with corresponding coordinates chosen from one of thereduced-scale images or the 1/1 scale image. The image from which saidpixel is taken is chosen as a function of the orientation of thecorresponding unit surface.

[0022] Thus a two-dimensional image transported in a stream of data issubjected to successive scale reductions to obtain a plurality of imagesof reduced size. The two-dimensional image to the original scale can bestored in memory with the reduced-size images. A three-dimensional imageis generated in real time from a frame provided with unit surfaces, andpixels chosen from said image to the original scale and saidreduced-size images. The pixels are chosen as a function of theorientation of a unit surface to which the pixel is to be assigned.

[0023] The present invention also provides a system for processingdigital images including a digital terminal, a central unit, a memorysystem and a display unit. The digital terminal includes a bit blockprocessor module adapted to reduce the size of a two-dimensional image.The memory system is adapted to store a plurality of reduced size imagesderived from a two-dimensional image, and said reduced sizes aredifferent from each other. An auxiliary module associated with thedigital terminal generates a three-dimensional image from a frameprovided with unit surfaces and from pixels chosen from saidtwo-dimensional image and the reduced size images from the bit blockprocessing module. The choice is made as a function of the orientationof a unit surface to which said pixel is be assigned. Saidthree-dimensional image is sent to the display unit.

[0024] In one embodiment of the invention the auxiliary module has adisplay output connected to the display unit.

[0025] The auxiliary module advantageously has a display outputconnected to the digital terminal.

[0026] The central unit is advantageously able to determine theorientation of the unit surface to which said pixel is to be assigned.In other words, a 3D rendition circuit is introduced into a digitalterminal to obtain real time images of high quality.

[0027] The invention enables a high-quality three-dimensional effect tobe offered that is free of the aliasing artifacts that are a function ofthe orientation of the unit surfaces of the array defining the structurerepresented. Not only the full-scale image is used, but also the imagesto scales of ½, ¼, etc., down to the scale of one pixel per unitsurface. The use of reduced scale images enables filtering to be locallyadapted as a function of the orientation of each unit surface relativeto the display plane.

[0028] For example, a unit surface whose normal is at an angle to thedisplay plane whose sine is from 1 to 0.75 is allocated a pixel from the1/1 scale image. If the sine is from 0.75 to 0.375, the pixel assignedis from the ½ image. If the sine is from 0.375 to 0.1875, the pixelassigned is from the ¼ scale image. If the sine is from 0.1875 to0.09375, the pixel assigned is from the ⅛ scale image. If the sine isless than 0.09375, the pixel assigned is from the {fraction (1/16)}scale image.

[0029] The present invention will be better understood and otheradvantages will become apparent on reading the following detaileddescription of one embodiment of the invention, which is given by way ofnon-limiting example only and illustrated by the accompanying drawings,in which:

[0030]FIG. 1 shows an example of a full-scale two-dimensional image;

[0031]FIG. 2 shows a corresponding example of reduced scaletwo-dimensional images;

[0032]FIG. 3 is a diagram showing cylindrical three-dimensionalreconstruction; and

[0033]FIG. 4 is a diagram of a system according to one aspect of theinvention.

[0034]FIG. 1 shows one example of a full-scale two-dimensional image,here comprising 512×256 pixels.

[0035]FIG. 2 shows the full-scale image 1, a ½ scale image 3, a ¼ scaleimage 4, a ⅛ scale image 5, a {fraction (1/16)} scale image 6 and a{fraction (1/32)} scale image 7 placed in a predetermined formatrepresented by the rectangle 2. It can be seen that, on changing fromany of these images to the image of the immediately next smaller format,the scaling applies both to the width and to the height of the image,and that the quantity of pixels, i.e. the quantity of data, is reducedby a factor of 4 each time on changing from a scale of ½ to the power xto the scale of ½ to the power x+1.

[0036] The reduced format images 3 to 7 are generated from the image 1by filtering, which can be of the type that takes one pixel in four, forexample one row in two and one column in two, or of the type thatcalculates an average of adjacent pixels, or a weighted average of agreater number of pixels. Thus the image 3 is generated from FIG. 1 byfiltering.

[0037] The image 4 is generated from the image 3 by filtering, which canbe effected four times faster than the filtering of FIG. 1, because thenumber of pixels is four times less, and so on. It follows that the timefor filtering the images 3 to 6 to generate the respective images 4 to 7is of the order of one third of the time to filter the image 1 togenerate the image 3. If a higher scale reducing factor is used, forexample a factor of 3, said time is even further reduced, for example toa value of the order of 12.5%. Here the images 1 and 3 to 7 are disposedone after the other from the first column and the first row left free byimages of higher rank. The addressing for recovering said images istherefore particularly simple.

[0038] The video is then transmitted in accordance with the standardITU-RBT.656 video protocol to a 3D rendering operator similar to thegraphics accelerators used in PCs.

[0039] However, for optimum memory storage, so that the memory space canbe reduced, image 4 could be placed under image 1 and to the right ofimage 3, in other words from the first row left free by image 1 and thefirst column left free by image 3, and so on.

[0040] In other words, a 3D controller is added to a digital terminal toobtain the maximum benefit from it, in terms of image quality, when itmanipulates animated video (for example: program changes with 3Dtransition effects: page turning, explosion, tearing, etc).

[0041]FIG. 3 shows a cylinder 8 with a vertical axis in the plane of thedrawing onto which pixels are projected to obtain a three-dimensionalimage. The cylinder is divided into unit rectangles whose visiblesurface area is at a maximum when the rectangle is in the plane of thedrawing, i.e. parallel to the screen on which the image is displayed, inother words when the normal of the rectangle is directed toward the userof the image, and increasingly small as the angle of the rectanglerelative to the rectangle previously described increases, in other wordsas the normal of the rectangle approaches a plane parallel to thedisplay screen.

[0042] Areas 9 to 12 are defined on the surface of the cylinder 8 inaccordance with this criterion, the area 9 comprising unit rectanglessubstantially parallel to the plane of the display screen and area 12comprising unit rectangles substantially orthogonal to said displayplane. A distribution with a greater number of areas could have beenprovided. However, to clarify the image, they are limited here to four.Area 9 is filled with pixels to the 1/1 scale from image 1. Area 10 isfilled with pixels with coordinates corresponding to the ½ scale ofimage 3. Area 11 is filled with pixels with coordinates corresponding tothe ¼ scale of area 4, and area 12 is filled with pixels withcoordinates corresponding to the ⅛ scale from area 5, and so on if agreater number of areas is provided.

[0043] The processor system shown in FIG. 4 includes a digital terminal15 with an input port 16, a video output port 17 and communication ports18, 19 and 20, a central unit 21 able to communicate with the port 18 ofthe digital terminal 15, and an auxiliary three-dimensional processorunit 22 able to communicate with the central unit 21 and with thedigital terminal 15 via the input port 19 and the output port 20.

[0044] The output port 17 of the digital terminal 15 is usuallyconnected to a video screen 23, for example a television monitor, inparticular of the kind with interlaced scanning. The auxiliary unit 22can optionally be provided with a VGA output, not shown.

[0045] From a stream of data relating to two-dimensional images arrivingat the input port 16, the digital terminal 15 generates an image file ofthe FIG. 2 type and sends it via the output port 20 to the auxiliaryunit 22. The central unit 21 receives the incoming stream of data viathe digital terminal 15 and performs the geometrical calculation; inother words, for each unit surface, it calculates the angle of thenormal vector to said surface and sends it to the auxiliary unit 22.

[0046] The auxiliary unit 22 then assembles the three-dimensional imagein the manner explained with reference to FIG. 3. Knowing the angle ofthe normal vector to a unit surface, the auxiliary unit 22 determineswhich reduced or non-reduced image to use and assigns the correspondingpixel to said unit surface. The three-dimensional image ready fordisplay is sent back to the digital terminal 15, which sends it via itsoutput port 17 to the screen 23.

[0047] The invention applies particularly to generating transitioneffects between two applications, for example a page-turning effect.This imparts a very high image quality to a broadcast video sequence,i.e. one not stored beforehand by the user, comparable to that of videogames running on dedicated consoles, for which the images are stored ina memory of the console.

[0048] A decoder 7, for example an MPEG decoder and/or a decoderdedicated to descrambling, can be provided on the upstream side of thedigital terminal 15 from FIG. 4, and itself preceded by a demodulator.

[0049] To be more specific, the digital terminal 15 would be providedwith a bit block processing module 24, called a “blitter” (FIG. 4), foreffecting the filtering needed to generate images on a reduced scale.Reduced format images from the blitter output are placed in a memorywith particular addressing arrangements, for example so that images 3 to7 are stored as shown in FIG. 2.

1. A method of processing digital images wherein: a plurality of imagesof reduced size are generated from a two-dimensional image, said reducedsizes being different from each other, said images of reduced size aretransmitted to a 3D accelerator encapsulated in accordance with apredetermined structure, said reduced size images are stored in memoryby extracting said images from the encapsulation structure, and athree-dimensional image is generated in real time from a frame providedwith unit surfaces and from pixels chosen from said image and saidreduced size images as a function of the orientation of a unit surfaceto which said pixel is to be assigned.
 2. A method according to claim 1,wherein a reduced image is generated from the whole of an animatedtwo-dimensional image.
 3. A method according to claim 1 or claim 2,wherein said reduced sizes are in geometrical progression.
 4. A methodaccording to any preceding claim, wherein storage in memory is effectedin accordance with a particular writing order.
 5. A method according toany preceding claim, wherein reading in memory is effected in aparticular order.
 6. A method according to any preceding claim, whereinsaid reduced size images are formatted inside the format of said imageso that all of said reduced size images occupy less than or the samespace as said image.
 7. A method according to any preceding claim,wherein the three-dimensional image is stored in memory.
 8. A methodaccording to any preceding claim, wherein the three-dimensional image issent to a display system.
 9. A system for processing digital imagesincluding a digital terminal (15), a central unit (21), a memory systemand a display unit (23), characterized in that the digital terminalincludes a bit block processor module (24) adapted to reduce the size ofa two-dimensional image and an auxiliary memory (22) associated with thedigital terminal for generating a three-dimensional image from a frameprovided with unit surfaces and from pixels chosen from saidtwo-dimensional image and the reduced size images from the bit blockprocessing module as a function of the orientation of a unit surface towhich said pixel is be assigned, in which system the memory system isadapted to store a plurality of reduced size images (4 to 7) derivedfrom a two-dimensional image (1), said reduced sizes are different fromeach other, and said three-dimensional image is sent to the display unit10. A system according to claim 9, characterized in that the auxiliarymodule has a display output connected to the display unit.
 11. A systemaccording to claim 9 or claim 10, characterized in that the auxiliarymodule has a display output connected to the digital terminal.
 12. Asystem according to any of claims 9 to 11, characterized in that thecentral unit is able to determine the orientation of the unit surface towhich said pixel is to be assigned.